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• Methods : ex post, pre-post, and 
experimental methods 

• Ex post evaluation : weaknesses and 
opportunities (Tanzania and Zambia) 

• Enduring questions : cost 
effectiveness and tangible impact of 
D/G assistance 



Why Experimental? 



• Permits causal conclusions and 
testing of theories 

• Isolates the intervention; does not 
conflate with spurious changes 
(positive or negative) 

• Pressing need for evidence in this 
field 



Evaluation 


Finding 


Uganda Health 
Project 


Community monitoring via scorecards resulted in improved 
wait times and quality of care (increase in immunization rate 
and decrease in child mortality) 


India Education 
Project 


Providing communities with scorecards rating school qualities 
did not result in improved learning outcomes, teacher effort, 
or community participation 


Indonesia Community 
Driven Development 
Project 


Citizen participation in decision making related to local 
government expenditures did not lead to decreases in 
corruption (as compared to top-down audit reforms) 



$24 million program administered by USAID 



• Duration of activities ranges from 2 to 3 years; 
started in 2009 

• Activities : 

— Increase civic participation and train government staff 
on improving community outreach in 30 districts 

— Install over 200 police complaints/commendation 
boxes 



Rwanda IE 



• Outcome of interest : increased perception 
that local budgets reflect their priorities; 
increased participation; improved perception 
of police accountability/responsiveness 

• Evaluation Method : Clustered randomization 
approach; adapted to project realities 
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Tanzania Program Facts 



• $11 million program administered by USAID 

• 2-year program ended in late 2008 

• Primary objectives (\|/ corruption and^ ROL): 

1. Train prosecutors and investigators 

2. Build civil society and media capacity, 
particularly to track public expenditures 

3. Conduct audits of procuring entities 



Tanzania 
Basic Evaluation Facts 



• Ex post evaluation of program results 
using quantitative and qualitative 
methods 

• Limited monitoring data 

• Look for evidence of impact to reject 
the null hypothesis 



Tanzania Threshold Program 
Ex Post Evaluation Methods 



• Police/prosecutors - quantitative 
analysis of phased training 

• Procurement/PETS- compare audit 
findings from before and after the 
project 

• Media - article quality analysis 



Zambia Threshold Program at a Glance 



• $22.75 million program administered by USAID 



• 2-year program ended in early 2009 



• Primary objectives: 

1. Reduce administrative corruption 

2. Improve business environment 



Millennium 

CI IAU.ENGE CORPORATION 

UNITED STATES OF AMERICA 



11 



Zambia's Threshold Progra m: Organization 



Government of Zambia 




Chemonics 



Ministry of Lands 1 1 ZRA 
1.6 m 



Int'l Lands Systerrj I 

$234,880 




Economic Freedom 





Zambia Business 
Forum $326,000 



Basic Evaluation Facts 



• Ex post qualitative evaluation of program 
results and lessons learned one year after 
program end 

• No counterfactual 

• Limited baseline and monitoring data 

• Look for evidence of impact to reject the 
null hypothesis 
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• Two Activities: 

— Creation of four one-stop shops 

— Automation of manual procedures 

• Two Goals: 

— Reduce days to register a business 

— Improve transparency 

• Reported Findings : 

— Decrease in registration time 

— Increase in satisfaction with services 
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• Evaluation (some recall data): 

- Exit survey of 40 applicants in Lusaka, 15 in 
Livingstone; 

- In-office survey of 15 "bulk filers"; 

- Interviews with GoZ and implementers 

• Baseline data : Survey of 77 applicants based 
on PACRO lists by implementer 
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Same day < 1 week < 2 weeks 

Time to Register a Business (2010) 
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Findings in Lusaka and Livingstone 
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Drawing Inferences 



• Pre- and post-comparison: 

- Project Survey 2006: 28 days to register a business 

- 2010 Project Survey: 80% responded <2 weeks 

• 69% of all respondents said applications for any 
procedures completed in < 1 week 

• Of 16 firms with prior PACRO experience, 11 
agreed that service is faster 
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Reducing Opportunities for Corruption 



Activities : 

— Automating procedures in Ministry of Lands, 
the Revenue Authority and Dept of Immigration 

— Building capacity of the ACC 

— Streamlining border management 

Findings : 

— Automation: Mixed results 
-ACC: Little impact 

— Border management: Mixed results MIL ^ 
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Evaluation (using recall, where relevant): 

- Exit survey of 35 immigration applicants in Lusaka; 

- Exit survey of 38 land clients; 

- Exit survey of 30 revenue authority registrants; 

- 9 firms enrolled in Customs Accredited Client Program 
(CACP) 

- In-office survey of 17 bulk filers; 

- Interviews with GoZ and implementers. 

Baseline data by implementer : 

- Surveyed 144 customers for Lands ; 156 Immigratio 
applicants; 89 Revenue Authority registrants; 121 
border agents 
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Inferences on Immigration 



Q: How would you compare levels of corruption 
now to before the ZIP changes? 

A: 53% responded "better", 47% responded "no 
opinion" or "worse" 



Q: How would you compare the level of customer 
care on recent visit compared with your first visit? 
A: 73% responded "better" 20% responded 
"same" 7% responded "worse" ^ 
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Inferences on Land 



Q: How would you compare levels of corruption 
now to before the ZIP changes? 

A: 76% responded "no opinion"; 6% responded 
the "same"; 18% responded "better" 

Q: How would you compare application processing 
times to before the ZIP changes? 

A: 23% responded slower; 24% responded 
"same", "slower"; 41% responded "faster" 
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Are bribes required for service at ZRA 
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Great deal Sometimes Never No opinion 



Inferences on Customer Service 

Charters 




1. Identify institutions with mid-level reformers 

2. Narrow program focus to facilitate oversight 

3. Set milestones to de-politicize termination or 
re-programming decisions 

4. Integrate IT after regulatory reforms 

5. Collect structured baseline and midterm 

data to provide time series and account for 
possible seasonality; integrate phased roll- 
out of program as part of design 



